1. The Field of the Invention
This invention relates to methods and systems for computerized learning, response, and evaluation. More specifically, this invention relates to methods and systems for organizing and presenting information to a student and allowing the student a great deal of flexibility in determining the testing and evaluation procedure.
2. The Relevant Technology
Testing has typically held a central role in the learning process. In educational settings, an individual is first presented with information that is to be learned and, in order to measure the amount of information learned, a test is given. Testing has evolved into a specialized field. Many theories have been developed in order to both develop tests designed to measure a particular level of achievement or mastery and to interpret the results of such a test. Thus, standardized tests have been developed to measure such things as Intelligence Quotient, personality traits, fitness to practice a profession such as law, medicine, contracting or the like, aptitude for success in a specific environment, and mastery of individual skills.
Historically, tests have been administrated on either an individual basis, such an oral or written examination, or on a group basis, such as a test administered in an educational setting where students record their answers on paper which are to be evaluated at a later time. One of the advantages of a written test is the efficiency in testing a large number of individuals. Unfortunately, a written test administered in such a manner lacks the ability to provide immediate feedback on an individual basis. Thus, it is generally recognized that testing in a traditional educational setting represents a tradeoff between efficiency and the ability to provide individual feedback.
It is generally unreasonable to expect that an individual test would be developed for each individual. Thus, in order to preserve the ability of the test to measure an individual's aptitude, care must be taken not to disclose the answers to a test. Likewise, where the test is designed to measure individual performance, care must be taken to insure that each individual takes the test based on his or her own knowledge.
In order to prevent cheating and to minimize the probability of an individual test score being influenced by others, multiple tests may be developed and administered simultaneously so that each individual is, in effect, taking a different test. Problems associated with this methodology, however, generally prevent its implementation. The effort and expense of developing and grading multiple tests generally limits this methodology to very specialized applications.
Many educators believe that measurement of performance and feedback is critical in the learning process. While written tests may be an efficient way to measure performance, they are largely inefficient in providing adequate feedback for learning. Generally there is a large lag time between the time when the test is taken and when the results of the test are returned. During this lag time, new information is usually presented. Thus, when the tests are returned, the focus of both the material and the individual has shifted. Thus, relatively few students will use such a test to evaluate their weaknesses and return to prior material in order to strengthen those weaknesses.
In order to overcome this limitation, some courses of study are structured with many smaller tests interspersed throughout the material. While this generally improves the potential for feedback, still such a method is relatively inefficient. Further, it dramatically increases the workload of an instructor or teacher.
In order to enhance the learning environment, many educators realize that the optimal form of instruction would be a personalized instructor for each student or individual. This, however, is generally impractical in most learning environments. In order to provide a better learning environment that more closely approaches the ideal environment of one instructor for every student, some educators are turning to computers. Through the use of computers, learning programs may be developed which provide instruction and feedback virtually simultaneously. For example, a general course of instruction may be presented to an individual after which the computer can query the individual regarding the principles just learned. The computer can then tally the score and provide the score to the individual. This allows the individual to return to information not learned and again review the material. Another advantage of computers is that they allow the pace of instruction to be varied according to the ability of the individual to learn. Furthermore, computers can be used to enhance personal learning outside the traditional educational environment. Computers thus hold great potential for enhancing the learning environment.
Current utilization of computers as part of the learning environment take their modes and methods of operation from current teaching techniques. Most computer programs designed to aid in the learning process first present a section of information and then test the individual based on the information presented. The structures of these programs are generally organized in a pre-set or pre-defined manner. Thus, like text books generally utilized in educational settings, the program presents a chapter of information and then quizzes the user on the information contained in that chapter. The next chapter is then presented, if any, along with the associated test. Programs which are organized to present information in this manner, are generally very inflexible and do not allow the user to determine in which order the information will be presented. Although this has the advantage of presenting information in an order which has been shown by long experience to be the "best" for learning the information, it forces the user to review information that may already be known.
In order to provide more flexibility, some programs allow the user to select which chapters will be presented. Thus, the user is allowed to determine the basic ordering of the chapters of information. While this allows the user some degree of flexibility in the process, it still deals with information in relatively large units. Thus, the presentation of information still remains relatively fixed and ridged. Concepts from different units or chapters must be tied together in the mind of the user. Similarly, because the tests are presented after each unit, no overview test is provided. It would thus be an advancement in the art to provide a method to allow the user more control not only over the order and sequence of presentation but also the focus and scope of the tests administered.
In an attempt to remedy this shortcoming, some programs are also provided with a global or overview test. These tests allow a user to be tested on the entire subject matter. Because of the general nature of these tests, a user must generally review all material in the program before these tests can be utilized. Thus, there still remains an inability to test only on the information that has been presented to a user.
Another potential problem with these types of programs is the inability of the program to vary its behavior when presenting test questions. Once an individual has worked through a lesson or chapter, the set of questions at the end are generally fixed. Thus, an individual who must review the material several times in order to learn it, may learn the sequence of test questions. The test then becomes less a measure of the actual knowledge possessed by the user and more a measure of the ability of the user to memorize the sequence and order of test questions. In order to prevent this occurrence, many programs have built into them a random component. This component presents the test questions in a random order. Memorization of the test questions by an individual is thus more difficult. While this has the advantage of providing a little different look each time the program is run, it still does not totally solve the problem.
In an attempt to strengthen this part of the program, certain manufactures have created a larger set of questions. The test would then ask a subset of these questions in a random order. In this way, not only is the order of test questions randomized, but also the selection of the test questions themselves. Thus, a user reviewing the same material on two different occasions, may not be presented with the same questions. While this provides an even greater degree of randomness so that the tests are different each time they are taken, care must be utilized to insure that each question is asked only once. Furthermore, the effectiveness of this method depends on the number of questions available for use relative to the number of questions which are asked during each test.
This problem raises one of the limitations of these types of programs. The effectiveness of the test and evaluation section of the program is generally dependent on the number of questions which can be asked of an individual. During development of these programs, it is often very difficult and time consuming for an individual to generate a pool of questions for use during testing. Each question must be authored an then programmed into the computer so that it can be asked at the appropriate time. The larger the number of test questions, the harder the program is to develop. Thus, many of these type of programs utilize relatively simple testing methodologies in order to minimize the total number of test questions that must be developed.
Speed and complexity of development are always paramount considerations when crafting these types of learning programs. Development of these types of programs generally begins by obtaining information. After information is obtained, the information must be sorted into logical categories for presentation. This process can be illustrated by a glance through a typical classroom textbook. In the textbook, information is organized or grouped by chapters to provide a logical and coherent flow for the individual utilizing the textbook. In much the same way, information is organized and structured into chapters or units or lessons to facilitate this logical and orderly flow. For each chapter or lesson, if testing is desired, then a pool of test questions must be drafted and developed. The answers to these questions must also be developed. The programmer or developer then organizes the information so that it can be presented to the user. The test questions and answers are also organized so they can be presented to the user.
The time it takes to develop a learning program can be very significant. In order to minimize the development time, there is a tendency to minimize either the amount of information presented or the number of test questions presented. This, however, is sometimes not acceptable since the resulting product is insufficient to present the depth or breadth of information required. For example, only questions dealing with the narrow range of presented information may be asked. The student is not presented with a broad enough spectrum of information to allow the answering of analogous rather than specific questions. Currently, there does not exist a satisfactory method of reducing development time. While some authoring tools exist, these are generally structured so as to minimize the effort involved in presenting information to the user. Test questions and testing methods must still be developed through a very laborious process. Thus, it would be an advancement in the art to have a method which reduces the effort required to develop test questions.
One advantage of learning systems designed to test individuals on a sequence of materials is the ability to alter the number of questions being asked. For example, if the test is utilized for practice purposes, it would be desirable to ask fewer questions of an individual who clearly has mastered the subject and more questions of an individual who has not attained a high degree of mastery. Computers are capable of dynamically totaling the number of questions asked and determining whether further questions should be asked. Unfortunately, this capability is often limited by the number of questions available for a given chapter or lesson. Because such questions are often hard to develop, the number of questions available for a given chapter or subject is often limited. This is particularly true where a certain number of questions are to be reserved for a general test covering all information in the learning program.
Due to the difficulty in developing comprehensive learning programs, the testing questions are generally designed to cover only the material specifically presented. Because of the relative inflexibility and difficulty in developing broad-based learning programs, testing on information analogous to that which has been presented is generally difficult. In order to develop such a capability, an author must develop questions drawn to information only analogous to that which has been specifically presented. Because of the difficulty in developing analogous questions, such a capability is often not included in learning programs. It would thus be an advancement in the art to develop a simple method for testing on information analogous to that which has been presented.